Ever worked at a company (or on a codebase, or whatever) where it seemed like, no matter what the question was, the answer was written down somewhere you could easily find it? Most people haven t, sadly, but they do exist, and I can assure you that it is an absolute pleasure. On the other hand, practically everyone has experienced completely undocumented systems and processes, where knowledge is shared by word-of-mouth, or lost every time someone quits. Why are there so many more undocumented systems than documented ones out there, and how can we cause more well-documented systems to exist? The answer isn t people are lazy , and the solution is simple though not easy.

Why Johnny Doesn t Read When someone needs to know something, they might go look for some documentation, or they might ask someone else or just guess wildly. The behaviour look for documentation is often reinforced negatively, by the result documentation doesn t exist . At the same time, the behaviours ask someone and guess wildly are positively reinforced, by the results I get my question answered and/or at least I can get on with my work . Over time, people optimise their behaviour by skipping the look for documentation step, and just go straight to asking other people (or guessing wildly).

Why Johnny Doesn t Write When someone writes documentation, they re hoping that people will read it and not have to ask them questions in order to be productive and do the right thing. Hence, the behaviour write documentation is negatively reinforced by the results I still get asked questions , and nobody does things the right way around here, dammit! Worse, though, is that there is very little positive reinforcement for the author: when someone does read the docs, and thus doesn t ask a question, the author almost certainly doesn t know they dodged a bullet. Similarly, when someone does things the right way, it s unlikely that anyone will notice. It s only the mistakes that catch the attention. Given that the experience of writing documentation tends to skew towards the negative, it s not surprising that eventually, the time spent writing documentation is reallocated to other, more utility-producing activities.

Death Spiral The combination of these two situations is self-reinforcing. While a suitably motivated reader might start by strictly looking for documentation, or an author initially be enthused to always fully documenting their work, over time the reflex will be for readers to just go ask someone, because there s never any documentation! , and for authors to not write documentation because nobody bothers to read what I write anyway! . It is important to recognise that this iterative feedback loop is the natural state of the reader/author ecosystem, resulting in something akin to thermodynamic entropy. To avoid the system descending into chaos, energy needs to be constantly applied to keep the system in order.

The Solution Effective methods for avoiding the vicious circle can be derived from the things that cause it. Change the forces that apply themselves to readers and authors, and they will behave differently. On the reader s side, the most effective way to encourage people to read documentation is for it to consistently exist. This means that those in control of a project or system mustn t consider something done until the documentation is in a good state. Patches shouldn t be landed, and releases shouldn t be made, unless the documentation is altered to match the functional changes being made. Yes, this requires discipline, which is just a form of energy application to prevent entropic decay. Writing documentation should be an explicit and well-understood part of somebody s job description. Whoever is responsible for documentation needs to be given the time to do it properly. Writing well takes time and mental energy, and that time needs to be factored into the plans. Never forget that skimping on documentation, like short-changing QA or customer support, is a false economy that will cost more in the long term than it saves in the short term. Even if the documentation exists, though, some people are going to tend towards asking people rather than consulting the documentation. This isn t a moral failing on their part, but only happens when they believe that asking someone is more beneficial to them than going to the documentation. To change the behaviour, you need to change the belief. You could change the belief by increasing the cost of asking. You could fire (or hellban) anyone who ever asks a question that is answered in the documentation. But you shouldn t. You could yell RTFM! at everyone who asks a question. Thankfully that s one acronym that s falling out of favour. Alternately, you can reduce the cost of getting the answer from the documentation. Possibly the largest single productivity boost for programmers, for example, has been the existence of Google. Whatever your problem, there s a pretty good chance that a search or two will find a solution. For your private documentation, you probably don t have the power of Google available, but decent full-text search systems are available. Use them. Finally, authors would benefit from more positive reinforcement. If you find good documentation, let the author know! It requires a lot of effort (comparatively) to look up an author s contact details and send them a nice e-mail. The like button is a more low-energy way of achieving a similar outcome you click the button, and the author gets a warm, fuzzy feeling. If your internal documentation system doesn t have some way to close the loop and let readers easily give authors a bit of kudos, fix it so it does. Heck, even if authors just know that a page they wrote was loaded N times in the past week, that s better than the current situation, in which deafening silence persists, punctuated by the occasional plaintive cry of Hey, do you know how to ? . Do you have any other ideas for how to encourage readers to read, and for authors to write?

You may have heard that Uber has been under a bit of fire lately for its desires to hire private investigators to dig up dirt on journalists who are critical of Uber. From using users ride data for party entertainment, putting the assistance dogs of blind passengers in the trunk, adding a surcharge to reduce the number of dodgy drivers, or even booking rides with competitors and then cancelling, or using the ride to try and convince the driver to change teams, it s pretty clear that Uber is a pretty good example of how companies are inherently sociopathic. However, most of those examples are internal stupidities that happened to be made public. It s a very rare company that doesn t do all sorts of shady things, on the assumption that the world will never find out about them. Uber goes quite a bit further, though, and is so out-of-touch with the world that it blogs about analysing people s sexual activity for amusement. You ll note that if you follow the above link, it sends you to the Wayback Machine, and not Uber s own site. That s because the original page has recently turned into a 404. Why? Probably because someone at Uber realised that bragging about how Uber employees can amuse themselves by perving on your one night stands might not be a great idea. That still leaves the question open of what sort of a corporate culture makes anyone ever think that inspecting user data for amusement would be a good thing, let alone publicising it? It s horrific. Thankfully, despite Uber s fairly transparent attempt at whitewashing ( clearwashing ?), the good ol Wayback Machine helps us to remember what really went on. It would be amusing if Uber tried to pressure the Internet Archive to remove their copies of this blog post (don t bother, Uber; I ve got a Save As button and I m not afraid to use it). In any event, I ve never used Uber (not that I ve got one-night stands to analyse, anyway), and I ll certainly not be patronising them in the future. If you re not keen on companies amusing themselves with your private data, I suggest you might consider doing the same.

Unless you ve been living under a firewalled rock, you know that IPv6 is coming. There s also a good chance that you ve heard that IPv6 doesn t have NAT. Or, if you pay close attention to the minutiae of IPv6 development, you ve heard that IPv6 does have NAT, but you don t have to (and shouldn t) use it. So let s say we ll skip NAT for IPv6. Fair enough. However, let s say you have this use case:

1. A bunch of containers that need Internet access
2. That are running in a VM
3. On your laptop
4. Behind your home router!

For IPv4, you d just layer on the NAT, right? While SIP and IPsec might have kittens trying to work through three layers of NAT, for most things it ll Just Work. In the Grand Future of IPv6, without NAT, how the hell do you make that happen? The answer is Prefix Delegation , which allows routers to delegate management of a chunk of address space to downstream routers, and allow those downstream routers to, in turn, delegate pieces of that chunk to downstream routers. In the case of our not-so-hypothetical containers-in-VM-on-laptop-at-home scenario, it would look like this:

1. My border router (a DNS-323 running Debian) asks my ISP for a delegated prefix, using DHCPv6. The ISP delegates a /56¹. One /64 out of that is allocated to the network directly attached to the internal interface, and the rest goes into the pool , as /60 blocks (so I ve got 15 of them to delegate, if required).
2. My laptop gets an address on the LAN between itself and the DNS-323 via stateless auto-addressing ( SLAAC ). It also uses DHCPv6 to request one of the /60 blocks from the DNS-323. The laptop puts one /64 from that block as the address space for the virtual LAN (actually a Linux bridge) that connects the laptop to all my VMs, and puts the other 15 /64 blocks into a pool for delegation.
3. The VM that will be running the set of containers under test gets an address on the all VMs virtual LAN via SLAAC, and then requests a delegated /64 to use for the all containers virtual LAN (another bridge, this one running on the VM itself) that the containers will each connect to themselves.

Now, almost all of this Just Works. The current releases of ISC DHCP support prefix delegation just fine, and a bit of shell script plumbing between the client and server seals the deal the client needs to rewrite the server s config file to tell it the netblock from which it can delegate. Except for one teensy, tiny problem routing. When the DHCP server delegates a netblock to a particular machine, the routing table needs to get updated so that packets going to that netblock actually get sent to the machine the netblock was delegated to. Without that, traffic destined for the containers (or the VM) won t actually make it to its destination, and a one-way Internet connection isn t a whole lot of use. I cannot understand why this problem hasn t been tripped over before. It s absolutely fundamental to the correct operation of the delegation system. Some people advocate running a dynamic routing protocol, but that s a sledgehammer to crack a nut if ever I saw one. Actually, I know this problem has been tripped over before, by OpenWrt. Their solution, however, was to use a PHP script to scan logfiles and add routes. Suffice it to say, that wasn t an option I was keen on exploring. Instead, I decided to patch ISC DHCP so that the server can run an external script to add the necessary routes, and perhaps modify firewall rules and also to reverse the process when the delegation is released (or expired). If anyone else wants to play around with it, I ve put it up on Github. I don t make any promises that it s the right way to do it, necessarily, but it works, and the script I ve added in contrib/prefix-delegation-routing.rb shows how it can be used to good effect. By the way, if anyone knows how pull requests work over at ISC, drop me a line. From the look of their website, they don t appear to accept (or at least encourage) external contributions. So, that s one small patch for DHCP, one giant leap for my home network.

If you re someone who doesn t like Debian s policy of automatically starting on install (or its heinous cousin, the RUN or ENABLE variable in /etc/default/<service>), then running an init system other than systemd should work out nicely.

For some reason, I seem to end up writing software for very esoteric use-cases. Today, though, I think I ve outdone myself: I sat down and wrote a Ruby library to get and set process resource limits those things that nobody ever thinks about except when they run out of file descriptors. I didn t even have a direct need for it. Recently I was grovelling through the EventMachine codebase, looking at the filehandle limit code, and noticed that the pure-ruby implementation didn t manipulate filehandle limits. I considered adding it, then realised that there wasn t a library available to do it. Since I haven t berked around with FFI for a while, I decided to write rlimit. Now to find the time to write that patch for EventMachine Since I doubt there are many people who have a burning need to manipulate rlimits in Ruby, this gem will no doubt sit quiet and undisturbed in the dark, dusty corners of rubygems.org. However, for the three people on earth who find this useful: you re welcome.

Matt Palmer wrote a short but informative post about enabling DNS in a zone [1]. I really should setup DNSSEC on my own zones. Paul Wayper has some insightful comments about the Liberal party s nasty policies towards the unemployed [2]. We really need a Basic Income in Australia. Joseph Heath wrote an interesting and insightful article about the decline of the democratic process [3]. While most of his points are really good I m dubious of his claims about twitter. When used skillfully twitter can provide short insights into topics and teasers for linked articles. Sarah O wrote an insightful article about NotAllMen/YesAllWomen [4]. I can t summarise it well in a paragraph, I recommend reading it all. Betsy Haibel wrote an informative article about harassment by proxy on the Internet [5]. Everyone should learn about this before getting involved in discussions about controversial issues. George Monbiot wrote an insightful and interesting article about the referendum for Scottish independence and the failures of the media [6]. Mychal Denzel Smith wrote an insightful article How to know that you hate women [7]. Sam Byford wrote an informative article about Google s plans to develop and promote cheap Android phones for developing countries [8]. That s a good investment in future market share by Google and good for the spread of knowledge among people all around the world. I hope that this research also leads to cheap and reliable Android devices for poor people in first-world countries. Deb Chachra wrote an insightful and disturbing article about the culture of non-consent in the IT industry [9]. This is something we need to fix. David Hill wrote an interesting and informative article about the way that computer game journalism works and how it relates to GamerGate [10]. Anita Sarkeesian shares the most radical thing that you can do to support women online [11]. Wow, the world sucks more badly than I realised. Michael Daly wrote an article about the latest evil from the NRA [12]. The NRA continues to demonstrate that claims about good people with guns are lies, the NRA are evil people with guns.

• [1] http://tinyurl.com/lmmus8n
• [2] http://mabula.net/tbfw/=stop-the-unemployed
• [3] http://tinyurl.com/m7gb3cj
• [4] http://tinyurl.com/n3utw9c
• [5] http://tinyurl.com/nonvxt8
• [6] http://tinyurl.com/q9es57y
• [7] http://feministing.com/2014/09/11/how-to-know-that-you-hate-women/
• [8] http://tinyurl.com/p2vfmo3
• [9] http://tinyletter.com/metafoundry/letters/metafoundry-6-accident-blackspot
• [10] https://plus.google.com/+DavidHillJr/posts/fT3tNRVWL3o
• [11] http://tinyurl.com/q9ulocw
• [12] http://tinyurl.com/og7a9y7

Matt Palmer wrote a good overview of DNSSEC [1]. Sociological Images has an interesting article making the case for phasing out the US $0.01 coin [2]. The Australian $0.01 and $0.02 coins were worth much more when they were phased out. Multiplicity is a board game that s designed to address some of the failings of SimCity type games [3]. I haven t played it yet but the page describing it is interesting. Carlos Buento s article about the Mirrortocracy has some interesting insights into the flawed hiring culture of Silicon Valley [4]. Adam Bryant wrote an interesting article for NY Times about Google s experiments with big data and hiring [5]. Among other things it seems that grades and test results have no correlation with job performance. Jennifer Chesters from the University of Canberra wrote an insightful article about the results of Australian private schools [6]. Her research indicates that kids who go to private schools are more likely to complete year 12 and university but they don t end up earning more. Kiwix is an offline Wikipedia reader for Android, needs 9.5G of storage space for the database [7]. Melanie Poole wrote an informative article for Mamamia about the evil World Congress of Families and their connections to the Australian government [8]. The BBC has a great interactive web site about how big space is [9]. The Raspberry Pi Spy has an interesting article about automating Minecraft with Python [10]. Wired has an interesting article about the Bittorrent Sync platform for distributing encrypted data [11]. It s apparently like Dropbox but encrypted and decentralised. Also it supports applications on top of it which can offer social networking functions among other things. ABC news has an interesting article about the failure to diagnose girls with Autism [12]. The AbbottsLies.com.au site catalogs the lies of Tony Abbott [13]. There s a lot of work in keeping up with that. Racialicious.com has an interesting article about Moff s Law about discussion of media in which someone says why do you have to analyze it [14]. Paul Rosenberg wrote an insightful article about conservative racism in the US, it s a must-read [15]. Salon has an interesting and amusing article about a photography project where 100 people were tased by their loved ones [16]. Watch the videos.

• [1] http://www.hezmatt.org/~mpalmer/blog/2014/06/29/adventures-in-dnssec.html
• [2] http://thesocietypages.org/socimages/2012/05/05/a-case-against-the-penny/
• [3] http://www.molleindustria.org/blog/multiplicity/
• [4] http://carlos.bueno.org/2014/06/mirrortocracy.html
• [5] http://tinyurl.com/mrxjuff
• [6] http://tinyurl.com/k6abj26
• [7] https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile
• [8] http://www.mamamia.com.au/news/world-congress-of-families-conference/
• [9] http://tinyurl.com/n7sczw4
• [10] http://tinyurl.com/k3k36sq
• [11] http://tinyurl.com/k3k36sq
• [12] http://tinyurl.com/l42mckp
• [13] http://www.abbottslies.com.au/
• [14] http://www.racialicious.com/2009/12/21/and-we-shall-call-this-moffs-law/
• [15] http://tinyurl.com/karyryk
• [16] http://tinyurl.com/puydo94

If you ve noticed your chrome/chromium on Linux having problems since you upgraded to somewhere around version 35/36, you re not alone. Thankfully, it s relatively easy to workaround. It will hit people who keep their browser open for a long time, or who have lots of tabs (or if you re like me, and do both). To tell if you re suffering from this particular problem, crack open your ~/.xsession-errors file (or wherever your system logs stdout/stderr from programs running under X), and look for lines that look like this:

[22161:22185:0830/124533:ERROR:shared_memory_posix.cc(231)]
Creating shared memory in /dev/shm/.org.chromium.Chromium.gFTQSy
failed: Too many open files

And

[22161:22185:0830/124601:ERROR:host_shared_bitmap_manager.cc(122)]
Cannot create shared memory buffer

If you see those errors, congratulations! The rest of this blog post will be of use to you. There s probably a myriad of bugs open about this problem, but the one I found was #367037: Shared memory-related tab crash. It turns out there s a file handle leak in the chromium codebase somewhere, relating to shared memory handling. There s no fix available, but the workaround is quite simple: increase the number of files that processes are allowed to have open. System-wide, you can do this by creating a file /etc/security/limits.d/local-nofile.conf, containing this line:

* - nofile 65535

You could also edit /etc/security/limits.conf to contain the same line, if you were so inclined. Note that this will only take effect next time you login, or perhaps even only when you restart X (or, at worst, your entire machine). This doesn t help you if you ve got Chromium already open and you d like to stop it from crashing Right Now (perhaps restarting your machine would be a terrible hardship, causing you to lose your hard-won uptime record), then you can use a magical tool called prlimit. The prlimit syscall is available if you re running a Linux 2.6.36 or later kernel, and running at least glibc 2.13. You ll have a prlimit command line program if you ve got util-linux 2.21 or later. If not, you can use the example source code in the prlimit(2) manpage, changing RLIMIT_CPU to RLIMIT_NOFILE, and then running it like this:

prlimit <PID> 65535 65535

The <PID> argument is taken from the first number in the log messages from .xsession-errors in the example above, it s 22161. And now, you can go back to using your tabs as ersatz bookmarks, like I do.

$ sudo apt-get install -y leiningen
[...]
$ lein new scratch
[...]
$ cd scratch
$ lein repl
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.pom from repository central at http://repo1.maven.org/maven2
Transferring 5K from central
Downloading: org/sonatype/oss/oss-parent/5/oss-parent-5.pom from repository central at http://repo1.maven.org/maven2
Transferring 4K from central
Downloading: org/clojure/clojure/1.3.0/clojure-1.3.0.jar from repository central at http://repo1.maven.org/maven2
Transferring 3311K from central
[...]

Wait what? lein downloads some random JARs from a website over HTTP¹, with, as far as far I can tell, no verification that what I m asking for is what I m getting (has nobody ever heard of Man-in-the-Middle attacks in Maven land?). It downloads a .sha1 file to (presumably) do integrity checking, but that s no safety net if I can serve you a dodgy .jar, I can serve you an equally-dodgy .sha1 file, too (also, SHA256 is where all the cool kids are at these days). Finally, jarsigner tells me that there s no signature on the .jar itself, either. It gets better, though. The repo1.maven.org site is served by the fastly.net² pseudo-CDN³, which adds another set of points in the chain which can be subverted to hijack and spoof traffic. More routers, more DNS zones, and more servers. I ve seen Debian take a kicking more than once because packages aren t individually signed, or because packages aren t served over HTTPS. But at least Debian s packages can be verified by chaining to a signature made by a well-known, widely-distributed key, signed by two Debian Developers with very well-connected keys. This repository, on the other hand oy gevalt. There are OpenPGP (GPG) signatures available for each package (tack .asc onto the end of the .jar URL), but no attempt was made to download the signatures for the .jar I downloaded. Even if the signature was downloaded and checked, there s no way for me (or anyone) to trust the signature the signature was made by a key that s signed by one other key, which itself has no signatures. If I were an attacker, it wouldn t be hard for me to replace that key chain with one of my own devising. Even ignoring everyone living behind a government- or company-run intercepting proxy, and everyone using public wifi, it s pretty well common knowledge by now (thanks to Edward Snowden) that playing silly-buggers with Internet traffic isn t hard to do, and there s no shortage of evidence that it is, in fact, done on a routine basis by all manner of people. Serving up executable code to a large number of people, in that threat environment, with no way for them to have any reasonable assurance that code is trustworthy, is very disappointing. Please, for the good of the Internet, improve your act, Maven. Putting HTTPS on your distribution would be a bare minimum. There are attacks on SSL, sure, but they re a lot harder to pull off than sitting on public wifi hijacking TCP connections. Far better would be to start mandating signatures, requiring signature checks to pass, and having all signatures chain to a well-known, widely-trusted, and properly secured trust root. Signing all keys that are allowed to upload to maven.org with a maven.org distribution root key (itself kept in hardware and only used offline), and then verifying that all signatures chain to that key, wouldn t be insanely difficult, and would greatly improve the security of the software supply chain. Sure, it wouldn t be perfect, but don t make the perfect the enemy of the good. Cost-effective improvements are possible here. Yes, security is hard. But you don t get to ignore it just because of that, when you re creating an attractive nuisance for anyone who wants to own up a whole passel of machines by slipping some dodgy code into a widely-used package.

---

Gitolite is a popular way to manage collections of git repositories entirely from the command line it s configured using configuration stored in a git repo, which is nicely self-referential. Providing per-branch access control and a wide range of addons, it s quite a valuable system. In recent versions (3.6), it added support for configuring per-repository git hooks from within the gitolite-admin repo itself something which previously required directly jiggering around with the repo metadata on the filesystem. It allows you to chain multiple hooks together, too, which is a nice touch. You can, for example, define hooks for validate style guidelines , submit patch to code review and push to the CI server . Then for each repo you can pick which of those hooks to execute. It s neat. There s one glaring problem, though you can only use these chained, per-repo hooks on the pre-receive, post-receive, and post-update hooks. The update hook is special, and gitolite wants to make sure you never, ever forget it. You can hook into the update processing chain by using something called a virtual ref ; they re stored in a separate configuration directory, use a different syntax in the config file, and if you re trying to learn what they do, you ll spend a fair bit of time on them. The documentation describes VREFs as a mechanism to add additional constraints to a push . The association between that and the update hook is one you get to make for yourself. The interesting thing is that there s no need for this gratuitous difference in configuration methods between the different hooks. I wrote a very small and simple patch that makes the update hook configurable in exactly the same way as the other server-side hooks, with no loss of existing functionality. The reason I m posting it here is that I tried to submit it to the primary gitolite developer, and was told I m not touching the update hook [ ] I m not discussing this [ ] take it or leave it . So instead, I m publicising this patch for anyone who wants to locally patch their gitolite installation to have a consistent per-repo hook UI. Share and enjoy!

As my previous post suggested I'd been running a service for a few years, using Redis as a key-value store. Redis is lovely. If your dataset will fit in RAM. Otherwise it dies hard. Inspired by Memcached, which is a simple key=value store, redis allows for more operations: using sets, using hashes, etc, etc. As it transpires I mostly set keys to values, so it crossed my mind last night an alternative to rewriting the service to use a non-RAM-constrained service might be to juggle redis out and replace it with something else. If it were possible to have a redis-compatible API which secretly stored the data in leveldb, sqlite, or even Berkley DB, then that would solve my problem of RAM-constraints, and also be useful. Looking around there are a few projects in this area nds fork of redis, ssdb, etc. I was hoping to find a Perl Redis::Server module, but sadly nothing exists. I should look at the various node.js stub-servers which exist as they might be easy to hack too. Anyway the short version is that this might be a way forward, the real solution might be to use sqlite or postgres, but that would take a few days work. For the moment the service has been moved to a donated guest and has 2Gb of RAM instead of the paltry 512Mb it was running on previously. Happily the server is installed/maintained by my slaughter tool so reinstalling took about ten minutes - the only hard part was migrating the Redis-contents, and that's trivial thanks to the integrated "slave of" support. (I should write that up regardless though.)

I just made an account on yet another web service. On the suggestion of my password manager, I attempted to use the password W:9[$X*F . It was rejected because Password must contain at least one non-alphabet character, one lowercase letter, one uppercase letter . OK, how about Passw0rd ? Yep, that s fine. Anyone want to guess which of those two passwords is going to fall victim to a brute-force attack first? Go on, don t be shy, take a wild shot in the dark!

After dealing with the client side of the DNSSEC puzzle last week, I thought it behooved me to also go about getting DNSSEC going on the domains I run DNS for. Like the resolver configuration, the server side work is straightforward enough once you know how, but boy howdy are there some landmines to be aware of. One thing that made my job a little less ordinary is that I use and love tinydns. It s an amazingly small and simple authoritative DNS server, strong in the Unix tradition of do one thing and do it well . Unfortunately, DNSSEC is anything but small and simple and so tinydns doesn t support DNSSEC out of the box. However, Peter Conrad has produced a patch for tinydns to do DNSSEC, and that does the trick very nicely. A brief aside about tinydns and DNSSEC, if I may Poor key security is probably the single biggest compromise vector for crypto. So you want to keep your keys secure. A great way to keep keys secure is to not put them on machines that run public-facing network services (like DNS servers). So, you want to keep your keys away from your public DNS servers. A really great way of doing that would be to have all of your DNS records somewhere out of the way, and when they change regenerate the zone file, re-sign it, and push it out to all your DNS servers. That happens to be exactly how tinydns works. I happen to think that tinydns fits very nicely into a DNSSEC-enabled world. Anyway, back to the story. Once I d patched the tinydns source and built updated packages, it was time to start DNSSEC-enabling zones. This breaks down into a few simple steps:

1. Generate a key for each zone. This will produce a private key (which, as the name suggests, you should keep to yourself), a public key in a DNSKEY DNS record, and a DS DNS record. More on those in a minute. One thing to be wary of, if you re like me and don t want or need separate Key Signing and Zone Signing keys. You must generate a Key Signing key this is a key with a flags value of 257. Doing this wrong will result in all sorts of odd-ball problems. I wanted to just sign zones, so I generated a Zone Signing key, which has a flags value of 256. Big mistake. Also, the DS record is a hash of everything in the DNSKEY record, so don t just think you can change the 256 to a 257 and everything will still work. It won t.
2. Add the key records to the zone data. For tinydns, this is just a matter of copying the zone records from the generated key into the zone file itself, and adding an extra pseudo record (it s all covered in the tinydnssec howto).
3. Publish the zone data. Reload your BIND config, run tinydns-sign and tinydns-data then rsync, or do whatever it is PowerDNS people do (kick the database until replication starts working again?).
4. Test everything. I found the Verisign Labs DNSSEC Debugger to be very helpful. You want ticks everywhere except for where it s looking for DS records for your zone in the higher-level zone. If there are any other freak-outs, you ll want to fix those because broken DNSSEC will take your domain off the Internet in no time.
5. Tell the world about your DNSSEC keys. This is simply a matter of giving your DS record to your domain registrar, for them to add it to the zone data for your domain s parent. Wherever you d normally go to edit the nameservers or contact details for your domain, you probably want to do to the same place and look for something about DS or Domain Signer records. Copy and paste the details from the DS record in your zone into there, submit, and wait a minute or two for the records to get published.
6. Test again. Before you pat yourself on the back, make sure you ve got a full board of green ticks in the DNSSEC Debugger. if anything s wrong, you want to rollback immediately, because broken DNSSEC means that anyone using a DNSSEC-enabled resolver just lost the ability to see your domain.

That s it! There s a lot of complicated crypto going on behind the scenes, and DNSSEC seems to revel in the number of acronyms and concepts that it introduces, but the actual execution of DNSSEC-enabling your domains is quite straightforward.

DNSSEC is one of those things that have been on the I should poke at that sometime list, but today I decided to actually get in and play around with it. Turns out it s pretty trivially easy to setup, but I got bitten by a few gotchas during testing, that I thought worthwhile to document. Since I like to explain things, a lot of this article isn t how to so much as how does .

Configuring The Resolver First off, you need a resolver¹ that you can trust, and which implements DNSSEC validation. In theory, you could use your ISP s resolvers (if they re DNSSEC-enabled), or even Google s public resolvers (8.8.8.8 and 8.8.4.4), but it s really not recommended how much do you trust those, really? More importantly, how much do you trust the network path between them and you? Frankly, running a local on-host caching resolver is trivial, consumes next-to-no resources, and is highly recommended. It s what I did. I used unbound, finally casting free the venerable dnscache, which has served me well for many years. I chose unbound because it fits the do one small thing, and do it well philosophy that I cherish (unlike BIND), and because it s what all the cool kids appear to be using these days. Installing it was as simple as apt-get install unbound it s available in pretty much every distro, I believe. Out of the box, unbound in Debian wheezy comes configured for DNSSEC validation. Other methods of installation might not be so amenable, so just make sure your unbound.conf liiks like this, so that DNSSEC is enabled:

server:
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    val-log-level: 2

The val-log-level line isn t strictly required, but I found it useful to have unbound log info on validation failures to syslog. Season to taste. You also have to configure your system to actually use your new, shiny local resolver. Editing /etc/resolv.conf would, in an ideal world, work Just Fine, but between Magical Unhelpful Pixies (I m looking at you, NetworkManager) and DHCP, it often isn t that easy. Ultimately, though, you should configure your system so that the only resolver in /etc/resolv.conf is always 127.0.0.1. Anything else is a security hole waiting to happen. If you re using DHCP and a recent version of ISC dhclient, adding supersede domain-name-servers 127.0.0.1; to dhclient.conf should do the needful.

Weighing the (Trust) Anchor Whatever local resolver you choose, you need to tell it about trust anchors . This is a key (or keys) which are trusted to sign all the other data that the resolver will find. Like the rest of the DNS, DNSSEC is hierarchical, and you need to know who to start with before you can start. To resolve a name using DNS, you ask the root servers for the answer to a question (say, what is the address of www.example.com? ). The root server doesn t know the answer, but it knows who to ask, and it tells you where to go (typically, go ask the DNS servers for .com, which are X, Y, and Z ). Then you ask those servers, and they probably don t know, but they know who to ask, and they tell you to go ask those servers. This continues until you finally get a server which says, oh, yes, I know the answer to that! and hands over the information you re after. In DNSSEC, you get digital signatures that prove that the answers you get from each DNS server you ask are legit, and not forged. So when you ask the root servers, what is the address of www.example.com? , they answer, I don t know, but go ask DNS servers at addresses X, Y, and Z and by the way, you can trust this answer is legit because here s this signature you can check . To verify a signature is legit, you need to know that the key used to make the signature is trustworthy, and it wasn t just any ol key that made it similar to how an ink scribble on a page doesn t mean anything unless you know what the signature you re looking at should look like. DNSSEC has two ways to make sure you know who to trust. Firstly, those trust anchors I mentioned three paragraphs ago. These are the keys that the root servers use to sign the responses they send out. You tell unbound (or your local DNSSEC-enabled resolver of choice), if you see a signature provided by this trust anchor key, you can trust it to be legit . The second way to trust is key is to have someone you already trust say, oh, you can trust this key, too . This is done by providing more data in the answer that you get from the root servers. So the full answer you get from the root servers when you ask, what is the address of www.example.com? looks like this:

I don t know, but you should ask the DNS servers at addresses X, Y, and Z. You can trust their answers because they ll sign their responses with key K. Finally, you can trust that I m legit because of this signature I m providing you, generated from key T.

Assuming that T is the trust-anchor key you ve already told unbound to trust, it can then trust all of the information it got from that first question. When it then goes to DNS server X and asks, Hey, do you know the address of www.example.com? , that server can send back its own response saying:

Gee, shucks, I don t know, but DNS servers at P and Q can help you. You can trust them because they ll sign responses with key S. And just to show you I m on the level, check out this nifty signature I made, signed with key K.

Since a server we trust told us to trust key K, and then we got a response signed by key K that told us to trust key S, when we then go and ask server P and get a response signed by S, we re still demonstrably secure. Neat setup, right? Anyway, now you know all about trust anchors. How do you set it up? For unbound, it s capable of priming the trust anchor itself. It does this by retrieving a copy of the trust anchor from [a well-known HTTPS URL]((https://data.iana.org/root-anchors/root-anchors.xml), which provides a reasonable measure of security the trust anchor file should be safe as long as that HTTPS server is trustworthy. For the properly paranoid, you should determine an alternate means of verifying that you have the correct root key (a web search for DNSSEC root key attestation would probably be a good start).

Testing It s always handy to know that something s working as you expect. For something that s important for security, that goes double. So, how do you test that your DNSSEC-capable resolver is, indeed, working as designed? Well, firstly, you can test that the resolver, by itself, is working. This is simple: do a query for a zone known to have broken DNSSEC. Thankfully, some nice people (Comcast, specifically) have already set this up. You can test a resolver for DNSSEC-correctness with this command:

dig @127.0.0.1 www.dnssec-failed.org

If dig gives you back an IP address, something is broken. The correct, my DNSSEC works! response should be a status: SERVFAIL response. That s all well and good, but having a resolver that works isn t much good if nothing s actually using it. Your system configuration has probably got some other resolvers configured into it (such as those provided by a DHCP server), and you ll want to make sure those aren t getting used². The trivial test is to visit http://www.dnssec-failed.org in your browser. If that comes back with a page, then something s still wrong. A more user-friendly test of your DNSSEC configuration is provided by http://dnssectest.sidnlabs.nl/test.php, which will display a page either way, and give you a tick or a cross depending on whether you re DNSSEC-enabled or not. One thing to be wary of when you re doing browser tests is caching. I spent far too long trying to work out why unbound wasn t working , when in actual fact it was working fine, it s just that both my browser and local caching proxy (squid) were caching DNS requests³. So, even after I d correctly setup unbound, visiting the test sites was providing me with erroneous results. Fixing up the browser was simple enough. In Chromium, visiting chrome://net-internals#dns will show you what s being cached, what resolv.conf settings are being used, and allow you to clear the browser s resolver cache. Squid was similarly simple, although it took me a while to figure out that it was causing me grief (software that just works , being so rare, tends to slip your mind, and so I often forget that squid is actually doing me good). Restarting squid has the appropriate result (clears the DNS cache), but in the end I decided that having another layer of caching wasn t going to help significantly when I ve got an on-host DNS cache already, in unbound. So, I just turned the knobs for DNS caching in squid right down:

negative_dns_ttl 1 second
positive_dns_ttl 1 second

And voila! All was well.

In summary DNSSEC is cool. It s easy to get going with, and over time it will provide more and more practical benefits (I am a DANE fanboi). Get started with it now, and you ll be ahead of the curve.

---

1. A resolver is a program which takes a name, does all the DNS queries required to get the data associated with that name, and return it to whoever wanted to know. This can be a library which links to a client application, or it can be a separate program which receives DNS requests itself. There are usually both types of resolvers in use for a typical Linux system to resolve names. On every machine, in every program, there is the so-called stub resolver , which is part of libc, and does nothing except create DNS requests and send them to the addresses listed in resolv.conf. Also present, although not always running on every machine (typically your ISP or network admin has some running for everyone to share) are one or more recursive resolvers , which are separate programs which take DNS requests, do all of the queries themselves, and return the result to the stub resolver for presentation to the client program which made the request in the first place. You will see this material again.
2. You can configure dhclient to use your local resolver instead of the DHCP-provided ones by simply setting:

   supersede domain-name-servers 127.0.0.1;
   

   in your dhclient.conf file.
3. It also didn t help that I hadn t correctly knobbled dhclient the first time around, and so even when I thought I was only resolving against my local unbound server, it was still actually using the DHCP-provided resolvers

Since first posting my idea for an SSL co-op a couple of weeks ago, I ve gotten some positive feedback from people, and further thinking and research has convinced me that it is feasible to at least attempt it. As a result, I d like to announce the public unveiling of The SSL Co-op. It is intended to be a commercial, not-for-profit¹ organisation that issues widely-trusted certificates to members, for their use or for resale. Eventually, I d like the co-op to be a root CA in its own right, with its certificate trusted by all the browsers and other X.509-using applications out there, but that isn t something that s achievable immediately. At this stage, the co-op hasn t been formed, and I m looking for expressions of interest from individuals and organisations who would be interested in becoming members. If you fit that description, I d really appreciate it if you could fill out a short survey so I can get a better idea of what sort of scale the co-op will be operating at initially. This is the first step towards an interesting future, where there is more choice of provider for online identity verification. Exciting times.

---

Ten days ago, I blogged about (finally) generating a GPG key transition statement. As the title of this post suggests, I have received zero signatures. Have other people had any success with transition statements? Perhaps it s time to hit up Debian developers I know one-by-one To the IRCz!

I m probably the last person on the planet to get around to doing this, but I figure it s probably about time my venerable old 1024D key was put out to pasture. My transition statement (signed, as usual, by both the old and new keys) is available. If you signed my old key, and still trust that I am who I think I am, I d appreciate it if you could sign my new key and fling the signatures at me or the keyserver networks.

I ve been pondering co-ops, and their value in the world, lately. The thought has come to mind that a co-op which ran a Certificate Authority seems like an absolute no-brainer the members each contribute to the operational costs (servers, software, and the eye-watering auditing expenses) and in return get to issue all the domain-verified certificates they like (and pay reasonable rates for organization-verified certificates, and perhaps even EV certs). I know there are some superficially similar examples of this idea out there already. StartSSL captured my heart with their policy of only charging for what costs money (so you can get free DV certificates for personal use), but they made me sad when they charged for revocations (yes, I know revocations cost money to serve the CRL, but OCSP ain t free either ). CACert tickle my Free Software Fan-bone, being all about the freedom and community involvement, but lose some practicality points on the fact that they ve been trying to get their root certs into the browsers for a long time and haven t really gotten anywhere. So, assuming that DNSSEC (and hence DANE) doesn t become universally available any time soon, leaving the CA business model dead, buried, and worm-eaten, is the idea of a cooperative certificate authority of interest to anyone? Surely anyone who wished to have more control over their SSL certificates than you get with a reseller relationship, but couldn t justify becoming a root certificate holder themselves, would see the value of something of this nature?

This is the entireity of a (purportedly) HTML page I just got:

<script language="javascript">
  window.location = "http://example.com/obscured/to/protect/the/guilty"
</script>

To compound the pain, this didn t come from a site run by people who wouldn t be expected to know any better it s associated with a rather popular web-oriented test framework. So it should contain at least one person who might pipe up and say, WTF, don t do that! . I m up to about 7 things that are wrong with this. Anyone want to weigh in with their own enumeration of why this is shockingly bad?

Ruby has a reputation for being slow. But there s slow, and then there s three seconds to show the help for a fairly command line program . That s not slow, that s ridiculous. The gem cleanup command is the solution. Rubygems has a nice feature whereby you can have multiple versions of a gem installed at once. That s neat, because it allows programs with different gem version requirements to co-exist on the system. Unfortunately, over time, you can end up with many versions of a gem installed, as upgrades pull in ever-newer versions of all your gems. Couple this accumulation of cruft with a need for Rubygems to look at every one of them on every startup, and you can see how you could very quickly end up with a three second startup time. By running gem cleanup, you ll have the opportunity to nuke all the out-of-date versions of your gems. It takes into account dependencies, so it will at least ask you before nuking a gem that another gem absolutely depends on (I d prefer a command-line switch to say, Don t even think about uninstalling something that another gem depends on! , but you can t have everything). If you ve got non-gem things on your system that rely on out-of-date gems (anything using bundler, for instance), those will assplode the next time you run them, but you can always re-run bundle install to get just the specific versions of gems back you need. The end result of my little spring cleaning? Down from three seconds to three quarters of a second. A 75% improvement. Win!